Like trainer, like bot? Inheritance of bias in algorithmic content moderation

摘要

The internet has become a central medium through which `networked publics'express their opinions and engage in debate. Offensive comments and personalattacks can inhibit participation in these spaces. Automated content moderationaims to overcome this problem using machine learning classifiers trained onlarge corpora of texts manually annotated for offence. While such systems couldhelp encourage more civil debate, they must navigate inherently normativelycontestable boundaries, and are subject to the idiosyncratic norms of the humanraters who provide the training data. An important objective for platformsimplementing such measures might be to ensure that they are not unduly biasedtowards or against particular norms of offence. This paper provides someexploratory methods by which the normative biases of algorithmic contentmoderation systems can be measured, by way of a case study using an existingdataset of comments labelled for offence. We train classifiers on commentslabelled by different demographic subsets (men and women) to understand howdifferences in conceptions of offence between these groups might affect theperformance of the resulting models on various test sets. We conclude bydiscussing some of the ethical choices facing the implementers of algorithmicmoderation systems, given various desired levels of diversity of viewpointsamongst discussion participants.